Finding nuggets in documents: A machine learning approach

However, many text mining applications do not have adequate natural language processing ability beyond simple keyword indexing, and as a result, there are too many textual elements (words) included in the analysis. We argue that noun phrases as textual elements are better suited for text mining and could provide more discriminating power, than single words. Discourse representation theory (Kamp, 1981) and language learning of children (Snow & Ferguson, 1997) show that a document’s primary concepts are carried by noun phrases. Because noun phrase in a document are not equally important, we propose using them as candidates and identifying keyphrases from them. Document keyphrases are the most important topical phrases for a given document, and they address the main topics of that document. Our study proposes a Keyphrase Identification Program (KIP) to approach this problem by analyzing the composition of noun phrases. Keyphrases provide semantic metadata that can characterize documents and produce an overview of the content of a document. Keyphrases can be used in many text-mining related applications. If keyphrases are used in automatic text summarization, applications can extract sentences with more keyphrases or higher keyphrase scores. If keyphrases are used as document metadata, applications can use them to efficiently classify or cluster documents into different categories. They may be utilized to enrich the metadata of the results returned from a search engine. Another use is that some search engines implement interactive query refinement using keyphrases, and also use them as a way of browsing a collection. Last, but not least, keyphrases may be extracted from documents to construct a domain glossary or thesaurus. The previous studies of the various applications of keyphrases will be presented in the next section. Some documents, mostly scholarly papers, have a list of keyphrases provided by authors, but unfortunately, most documents do not have author-assigned keyphrases. Keyphrases can also be assigned manually by professional indexers. The indexers may choose phrases from the document text as keyphrases, or, more commonly, choose phrases from a predefined controlled vocabulary. However, manually assigning keyphrases to documents is costly and tedious, and the results Document keyphrases provide a concise summary of a document’s content, offering semantic metadata summarizing a document. They can be used in many applications related to knowledge management and text mining, such as automatic text summarization, development of search engines, document clustering, document classification, thesaurus construction, and browsing interfaces. Because only a small portion of documents have keyphrases assigned by authors, and it is timeconsuming and costly to manually assign keyphrases to documents, it is necessary to develop an algorithm to automatically generate keyphrases for documents. This paper describes a Keyphrase Identification Program (KIP), which extracts document keyphrases by using prior positive samples of human identified phrases to assign weights to the candidate keyphrases. The logic of our algorithm is: The more keywords a candidate keyphrase contains and the more significant these keywords are, the more likely this candidate phrase is a keyphrase. KIP’s learning function can enrich the glossary database by automatically adding new identified keyphrases to the database. KIP’s personalization feature will let the user build a glossary database specifically suitable for the area of his/her interest. The evaluation results show that KIP’s performance is better than the systems we compared to and that the learning function is effective.